UNITED STATES PATENT APPLICATION 



Title: 

DISTRIBUTED USAGE METERING OF MULTIPLE NETWORKED DEVICES 



Inventors: 
Benjamin T. Drucker 
Albert Y. Teng 



Docket No.: 42390.P13583 



Prepared by: 

Richard C. Calderwood 

Reg. No. 35,468 



"Express mail" label no. ET616076365US 



Docket No. 42390.P13583 



Page 1 



phone (408) 720-8598 



DISTRIBUTED USAGE METERING OF MULTIPLE NETWORKED DEVICES 

Background of the Invention 

Technical Field of the Invention 

The present invention relates generally to networking services, and particularly to a metering 
device for such. 

Background Art 

Network service providers desire to meter usage of their networks and servers, in order to 
provide load balancing, prevent fraud, enable accurate billing, and so forth. To date, there are two 
known metering models: ultra-fme-grain (UFG) and centralized. 

FIG. 1 illustrates a system 10 employing a UFG metering system. Each of the numerous 
customers 12 such as residences or businesses has a network device 14 which generates and receives 
network traffic, hi some systems, these network devices may be personal computers, cable television 
set-top boxes, or any other network devices. In the UFG model, each customer is provided with a 
metering device 16 networked to the one or more network devices at that customer's location. The 
metering devices and/or network devices are networked to a central service provider server 20 over 
one or more networking media using one or more networking protocol. 

Examples of networking media include digital subscriber line (DSL), coaxial cable, 
PhonePNA, HomePNA, fiber distributed data interface (FDDI), twisted pair, Ethernet wire, IEEE 
802.1 1 wireless, Bluetooth, HFC, GPRS, 3G, satellite, and so forth. Examples of networking 
protocols include TCP/IP, asynchronous transfer mode (ATM), AppleTalk, Token Ring, and so 
forth. The service provider server may, in turn, be connected to other networks and other servers. 
The service provider server performs networking services, data delivery, billing, and so forth, and 
also gathers and collates data reported by the multitude of metering devices 16. In many cases, the 
service provider server may be embodied as more than one server of different types, such as a 
primary server, a backup server, a billing system, a firewall, a front-end, a head-end, a back-end, a 
provisioning server, an encryption and authentication server, and so forth. 

FIG. 2 illustrates a system 22 employing a centralized metering system, in which each 
customer's location 12 is equipped with a networking device 14 but not a metering device. The 
metering is all done by the central service provider server 24. 
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Unfortunately, the UFG model is expensive - one metering device for each customer. Also, 
the service provider's server equipment must be able to deal effectively with interfacing directly to 
this large number of metering devices, which tends to raise the cost of the server equipment. 

And, unfortunately, the centralized model does not scale well. As more and more customers 
are added, the server's metering workload increases at least linearly. Maintenance increases 
accordingly. At some point, the server equipment may simply reach the limit of its metering ability, 
and it will not be possible to add any new customers without. replacing the server equipment with 
larger, more powerful, and more expensive servers. 

Furthermore, existing systems apply set metering rules and a fixed number of metrics at any 

given time. 

The U.S. Patent Application s/n 09/81 1,128 filed 3/16/2001 entitled "Gateway Metering and 
Bandwidth Management" shares a common inventor, Albert Teng, with this invention. That 
invention was directed to solving fraud by tracking multiple users of a single interface, by recording 
address ports on TCP/IP networks, for example. That invention defeated the ability of network 
address translation devices from hiding the true source of network traffic, which is a commonly 
employed fraud mechanism whereby e.g. two neighbors can both get network service while paying 
for only a single subscription. That invention has difficulty in certain circumstances, such as 
inaccurately identifying sources of network packets for applications that spawn multiple TCP 
sessions. 

What is desirable, then, is a metering apparatus, method, and system which is both less 
expensive than the UFG model and more scalable than the centralized model, and which relies on 
hardware identifications to identify traffic sources. 

Brief Description of the Drawings 

The invention will be understood more fully from the detailed description given below and 
from the accompanying drawings of embodiments of the invention which, however, should not be 
taken to limit the invention to the specific embodiments described, but are for explanation and 
understanding only. 

FIG. 1 illustrates an ultra-fine-grain metering system according to the prior art. 
FIG. 2 illustrates a centralized metering system according to the prior art. 
FIG. 3 illustrates a distributed, multi-device metering system according to one embodiment of 
this invention. 
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FIG. 4 illustrates a single-network embodiment of the metering device of this invention. 
FIG. 5 illustrates a dual-network embodiment of the metering device of this invention. 
FIG. 6 illustrates an embodiment of the metering device of this invention, configured to also 
provide hub/switch/router services. 

FIG. 7 illustrates one embodiment of the metering device of this invention. 

FIG. 8 illustrates one exemplary method of operation of the metering device of this invention. 

Detailed Description 

FIG. 3 shows a system 26 in which each customer 12 has one or more network devices 14 
coupled over suitable network media and protocol to the service provider's server 28. Metering is 
provided in a distributed usage (DU) metering manner, in which the metering is performed by a 
plurality of metering devices 30. Each metering device can be connected to more than one customer. 
Thus, the DU model employs fewer metering devices than the UFG model, but more than the single 
metering device (server) of the centralized model. 

As new users are added to the DU system, 

(a) the increased metering workload placed on the server is reduced by a factor of N, 
as compared to the centralized model, and 

(b) the increased expense of purchasing new meters is reduced by a factor of N, as 
compared to the UFG model, 

wherein N is the number of customers connected to a DU meter 30. N may, of course, be a 
variable number; it is not required that each DU meter have the same number of customers. 

In the UFG model, the average number of customers per metering device is 1 . In the 
centralized model, the average number of customers per metering device will typically be in the 
range of 512-10,000. In the DU model, the average number of customers per metering device will 
typically be in the range of 2-512; more commonly, it will be in the range of 4-128; and often, it will 

be in the range of 8-32. 

In this sense, "customers" can mean subscribing persons, or it can mean subscribing devices, 

or the like. 

In the DU model, the DU meters perform metering services for their respective customers, 
and then report their data or results to the central server, which may roll the data up into a single 
report or calculation. 
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There are various connection schemes whereby a metering device may be connected to a 
network. 

FIG. 4 shows a system in which the metering device 32 is coupled to a single network 
("network"), hi this embodiment, the metering device is coupled as a passive listening device, which 
simply monitors the network packets traveling to and from any and all of the network devices 14 
which are connected to that same network. 

FIG. 5 shows a system in which the metering device 34 serves as the connection point or 
gateway between one network ("network A") and another network ("network B"). In this 
embodiment, the metering device performs both gateway and metering services for the network 
devices 14 connected to one of the networks ("network A"). 

FIG. 6 shows a system in which the metering device 36 serves as the router or switch or hub 
between two or more networks ("network A" through "network D"). In this embodiment, the 
metering device performs both router/switch/hub and metering services for the network devices 14 
coupled to each of the networks, or coupled to a subset of the networks. 

FIG. 7 shows one exemplary embodiment of a metering device 40 ("Distributed Usage 
Meter") which incorporates the principles of this invention. The metering device 40 may be 
configured in any suitable configuration, such as one of those shown in FIGS. 4-6. The metering 
device includes one or more network interfaces 42a-d for connecting the network device to one or 
more corresponding networks 43a-d, which may use the same transport medium or different 
transport media, and which may use the same networking protocol or different networking protocols, 
as needed in the application at hand. 

The metering device may in some embodiments further include a switch or hub or router 
mechanism 44 coupled to the network interfaces, to perform switch/hub/router functionality. 

The metering device may in some embodiments also include a separate control interface 46 
for sending and receiving metering control commands, signals, and data. In some embodiments, the 
control interface may share a same physical networking medium with one or more of the attached 
networks, and the metering commands etc. maybe sent and received over one or more of the 
network interfaces, such as via the Simple Network Management Protocol (SNMP). In some 
embodiments, both the shared network/control interface and a dedicated control interface maybe 
employed, such as, for example, to permit remote control via the conventional network interface and 
local operator control via the dedicated control interface such as from a keyboard. In some 
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1 embodiments, the control interface may connect to a dedicated command link 47 which is distinct 

2 from the physical network media. 

3 The metering device may, in some implementations, include a display interface 48 for 

4 connecting the metering device over a display link 49 to a video or other suitable display mechanism 

5 (not shown), such as for use by a local operator, hi some embodiments, video and other output may 

6 instead be sent over the network interface and/or the control interface. In other embodiments, any or 

7 all of these may be present in combination. 

8 A packet header analyzer 50 performs the basic packet identification functionalities of the 

9 metering device. The packet header analyzer may, for example, analyze each network data packet to 

I o determine the identity of the network device which sent the packet, the identity of the network device 

I I which is to receive the packet, the communication protocol used by the packet, and so forth. In some 
1 ; 2 embodiments, the packet header analyzer may be built into the switch/hub/router, while in others it 
ffl may be standalone logic. 

U Coupled to the packet header analyzer is a mechanism for maintaining a detected device list 

f | 52, which keeps track of network devices that have sent and/or received network packets. This list 

|| may be maintained in any conventional manner, such as in a table, a linked list, and so forth. The list 

m 

hi mechanism may include memory and/or bulk storage for maintaining the list. 
|| Also present is a memory and/or bulk storage mechanism for storing weight definitions 54. 

!¥ These weight definitions comprise a collection of rules, formulas, Boolean values, logical operations, 

m or the like, for assigning or calculating a "weight" to one or more aspects of each packet analyzed by 

il the packet header analyzer. 

22 Characteristics by which the metering device may track packets, and per which the metering 

23 device may assign weights to those packets, include but are not limited to: communication protocol, 

24 packet size, time that the packet was sent, time that the packet was received, current average network 

25 throughput, current peak network throughput, total number of bytes transferred, total number of 

26 bytes transferred since some particular time or event, number of packets transferred that are in a 

27 given size range, traffic to or from particular addresses or ports or networks or sub-nets or network 

28 devices or categories of such, average or peak percentage of network utilization, average or peak 

29 number of TCP sessions open, average or peak traffic level of a particular protocol, percentage mixes 

30 of specified protocols among the current network traffic, or any other characteristic which the system 

3 1 designer deems worthy of metering. 
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A weight calculator 56 is coupled to the list of weight definitions, and performs the weight 
calculations, formulas, or the like. A packet weight history memory or storage 58 stores these 
weights for one, some, or all of the network devices whose packets are being analyzed. 

The weight definitions may be dynamically updated, either in response to internal logic (not 
shown) within the metering device, or in response to an externally received control command. For 
example, the network service provider may find it advantageous to track and bill by data type during 
the day, but by byte or packet count at night. Or, the network service provider may assign heavier 
metering weight to video during the day than it does at night or at times when network usage falls 
below some predetermined threshold. The skilled reader will readily appreciate that there are 
numerous ways in which a dynamically alterable set of weight definitions may be advantageous, and 
will be able to select a dynamic alteration scheme to suit the particular needs of the network system 
at hand. 

Similarly, it will be within the skill of the ordinary system designer to choose appropriate 
sizes, interfaces, speeds, protocols, and so forth, for these memories and/or bulk storage devices. 

The metering device further includes one or more clock mechanisms 60, such as a real time 
clock, a resettable elapsed time clock, a watchdog timer, and so forth. The data output by these 
clocks may be used by the weight calculator in performing its weighting operations, and may prove 
useful elsewhere, as well. 

The reader will further appreciate that the metering device shown in FIG. 7 is only by way of 
illustration, and that numerous differently-constructed embodiments of such devices will be 
appreciated in light of the teachings of this patent, when viewed in the context of designing a new 
metering device or a new network. Various enhancements and optional features have been omitted, 
for the sake of clarity. 

FIG. 8 illustrates one exemplary embodiment of a method (80) of operation of such a 
metering device. The reader may also wish to refer simultaneously to FIG. 7. Upon detection (82) of 
a newly-arrived packet or a next-to-be-analyzed packet, the packet header analyzer determines (84) 
the identity of the device sending the packet and the identity of the device receiving the packet. The 
metering device searches (86) the detected device list to determine whether these devices are already 
known to the metering device. If (88) the receiving device or the sending device has not previously 
been encountered (or has not been encountered since the detected device list was reset, or since that 
device's entry was flushed, etc.), that device is added (90) to the detected device list. 
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1 The weight calculator receives data from the packet header analyzer, regarding each of the 

2 characteristics upon which it will weight the packet, gets (92) the weight definitions from the weight 

3 definition list, and calculates (94) the respective weights for those indicated characteristics. The 

4 metering device then writes (96) these results to the packet weight history record(s) for the sending 

5 network device and/or receiving network device, as appropriate and in accordance with the weight 

6 definition rules. 

7 The operation, initialization, resetting, flushing, and so forth of the packet weight history are 

8 very application-dependent, and will be appreciated by the skilled reader when designing the 

9 networking system in light of these teachings. In some applications, it will be desirable for the 

10 history to be maintained over a long period of time, or perhaps even in perpetuity. In other 

1 1 applications, it will be desirable that some or all of the history be periodically reset to start afresh. 

12 For example, in some cases it may be beneficial to track, for each network device, a total byte count 
ffl sent to or from that network device since the billing period began, while resetting the percentage of 
|j network utilization metric every few minutes to allow for a more on-the-fly adjustment of bandwidth 
fjE allocation. 

II The skilled reader will also appreciate that the ratio of network devices to metering devices is 

17 application-dependent. In various system embodiments, ratios of 2:1, 4:1, 8:1, 12:1, 15:1, 100:1, or 

f| other ratios may be desirable, when balancing the cost of purchasing the required number of 

fp metering devices against the cost of scaling the network servers. Furthermore, the skilled reader will 

23 readily appreciate that it is not necessary that all segments of the network have the same metering 

if device ratio. For example, it may be found beneficial to use a different ratio for residential customers 

22 than for business customers, or a different ratio in town than in the countryside, or a different ratio in 

23 the LAN than on the WAN, and so forth. 

24 The reader should appreciate that drawings showing methods, and the written descriptions 

25 thereof, should also be understood to illustrate machine-accessible media having recorded, encoded, 

26 or otherwise embodied therein instructions, functions, routines, control codes, firmware, software, or 

27 the like, which, when accessed, read, executed, loaded into, or otherwise utilized by a machine, will 

28 cause the machine to perform the illustrated methods. Such media may include, by way of illustration 

29 only and not limitation: magnetic, optical, magneto-optical, or other storage mechanisms, fixed or 

30 removable discs, drives, tapes, semiconductor memories, organic memories, CD-ROM, CD-R, 

31 CD-RW, DVD-ROM, DVD-R, DVD-RW, Zip, floppy, cassette, reel-to-reel, or the like. They may 
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alternatively include down-the-wire, broadcast, or other delivery mechanisms such as Internet, local 
area network, wide area network, wireless, cellular, cable, laser, satellite, microwave, or other 
suitable carrier means, over which the instructions etc. may be delivered in the form of packets, 
serial data, parallel data, or other suitable format. The machine may include, by way of illustration 
only and not limitation: microprocessor, embedded controller, PLA, PAL, FPGA, ASIC, computer, 
smart card, networking equipment, or any other machine, apparatus, system, or the like which is 
adapted to perform functionality defined by such instructions or the like. Such drawings, written 
descriptions, and corresponding claims may variously be understood as representing the instructions 
etc. taken alone, the instructions etc. as organized in their particular packet/serial/parallel/etc. form, 
and/or the instructions etc. together with their storage or carrier media. The reader will further 
appreciate that such instructions etc. may be recorded or carried in compressed, encrypted, or 
otherwise encoded format without departing from the scope of this patent, even if the instructions 
etc. must be decrypted, decompressed, compiled, interpreted, or otherwise manipulated prior to their 
execution or other utilization by the machine. 

Reference in the specification to "an embodiment," "one embodiment," "some 
embodiments," or "other embodiments" means that a particular feature, structure, or characteristic 
described in connection with the embodiments is included in at least some embodiments, but not 
necessarily all embodiments, of the invention. The various appearances "an embodiment," "one 
embodiment," or "some embodiments" are not necessarily all referring to the same embodiments. 

If the specification states a component, feature, structure, or characteristic "may", "might", or 
"could" be included, that particular component, feature, structure, or characteristic is not required to 
be included. If the specification or claim refers to "a" or "an" element, that does not mean there is 
only one of the element. If the specification or claims refer to "an additional" element, that does not 
preclude there being more than one of the additional element. 

Those skilled in the art having the benefit of this disclosure will appreciate that many other 
variations from the foregoing description and drawings may be made within the scope of the present 
invention. Indeed, the invention is not limited to the details described above. Rather, it is the 
following claims including any amendments thereto that define the scope of the invention. 
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